{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import feedparser\n",
    "data = feedparser.parse('http://feeds.feedburner.com/37signals/beMH')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Signal v. Noise - Medium'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.feed.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Think back to the the last time you struggled mightily with\\xa0a\\xa0programming\\xa0problem. Did you share it with the\\xa0world?If you didn’t, that’s totally OK\\u200a—\\u200amost of us don’t! Why would we? Nobody enjoys admitting defeat, much less wanting to make a big deal out of\\xa0it.But kudos to you if you did share your struggles, because I bet you made a pretty big positive impact on someone. It very well may have inspired\\xa0them.I’m speaking from experience. Someone I respect recently did exactly this for me out of the blue. We were chatting a bit when they mentioned how they were struggling with some parts of Kotlin, just as I\\xa0was.What an astonishing revelation! I was surprised (and impressed) by this honesty. How could it be that this person, a great programmer whom I admire and has done amazing work, be struggling just like\\xa0me?!It’s strange\\u200a—\\u200alogically I know that of course everyone struggles and has rough patches. But in an era of highly polished tweets, blog posts, and conference talks, it’d be forgivable to think that programmers out there never struggle with their\\xa0work.But of course they do. Which is why when someone you respect shares their real-world struggles with you, it reinforces and crystalizes an important point: there’s no magic to anything we\\xa0do.These\\xa0vulnerable moments are a reminder that all of us are just programmers trying to do our best. That we all succeed basically the same way\\u200a—\\u200aby working hard,\\xa0struggling, learning, and keeping at it with determination.It also made me realize that while we often share our successes and expertise with the community, it’s much rarer that we humble ourselves and reveal our weaknesses. Don’t get me wrong\\u200a—\\u200ait’s amazing to be surrounded with smart people that are gracious enough to share their knowledge with us. A knowledgeable community is incredibly powerful.But it’s also incredibly powerful and inspirational to share your struggles. Doing so isn’t a sign of impostor syndrome\\u200a—\\u200ano, it’s a sign confidence, generosity, and honesty. I’ve been fortunate enough to be on the receiving end of such honesty more than once, and it always inspires me to keep at it and have confidence in\\xa0myself.So remember, we all struggle. If you’ve hit a rough patch today, don’t fret. There’s a good chance just about everyone else has too. Hopefully they’ll tell you about it\\xa0soon.✊If this article was helpful to you, please do hit the 💚 button below.\\xa0Thanks!We’re hard at work making the Basecamp 3 Android app better every day. Please check it\\xa0out!Your struggles can inspire others was originally published in Signal v. Noise on Medium, where people are continuing the conversation by highlighting and responding to this story.\\n \\n'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "test = data.entries[0].summary\n",
    "txt = re.compile(r'<[^>]+>').sub('',test)\n",
    "txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import urllib\n",
    "from bs4 import BeautifulSoup\n",
    "c = urllib.request.urlopen(\"http://www.bilibili.com\")\n",
    "soup = BeautifulSoup(c.read(), 'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<!DOCTYPE html>\n",
       "<html lang=\"zh-Hans\"><head><meta charset=\"utf-8\"/><title>哔哩哔哩 (゜-゜)つロ 干杯~-bilibili</title><meta content=\"bilibili是国内知名的视频弹幕网站，这里有最及时的动漫新番，最棒的ACG氛围，最有创意的Up主。大家可以在这里找到许多欢乐。\" name=\"description\"/><meta content=\"B站,弹幕,字幕,AMV,MAD,MTV,ANIME,动漫,动漫音乐,游戏,游戏解说,ACG,galgame,动画,番组,新番,初音,洛天依,vocaloid\" name=\"keywords\"/><meta content=\"webkit\" name=\"renderer\"/><meta content=\"IE=edge\" http-equiv=\"X-UA-Compatible\"/><meta content=\"gbRdPloQBZ\" name=\"baidu-site-verification\"/><link href=\"//s1.hdslb.com\" rel=\"dns-prefetch\"/><link href=\"//i0.hdslb.com\" rel=\"dns-prefetch\"/><link href=\"//i1.hdslb.com\" rel=\"dns-prefetch\"/><link href=\"//i2.hdslb.com\" rel=\"dns-prefetch\"/><link href=\"//static.hdslb.com\" rel=\"dns-prefetch\"/><link href=\"//static.hdslb.com/images/favicon.ico\" rel=\"shortcut icon\"/><link href=\"//static.hdslb.com/opensearch.xml\" rel=\"search\" title=\"哔哩哔哩\" type=\"application/opensearchdescription+xml\"/><link href=\"//s1.hdslb.com/bfs/static/phoenix/home/static/css/main.9477b13d89667cb7148dea16919b78f4.css\" rel=\"stylesheet\"/></head><body><div class=\"z-top-container has-banner\"></div><div id=\"home-app\"><app></app></div><div class=\"footer bili-footer\"></div><script type=\"text/javascript\">if(!+\"\\v1\"){\n",
       "        document.body.innerHTML = \"<div style='background: #00a1d6; text-align: center; padding: 10px 0; color: #fff;'>为了保护你的账号安全，bilibili已不支持IE8及以下版本浏览器访问，建议你升级到IE最新版本浏览器，或使用Chrome等其他浏览器。</div>\"\n",
       "      }</script><script src=\"//static.hdslb.com/js/jquery.min.js\" type=\"text/javascript\"></script><script charset=\"utf-8\" src=\"//static.hdslb.com/common/js/nav.js\" type=\"text/javascript\"></script><script charset=\"utf-8\" src=\"//static.hdslb.com/common/js/footer.js\" type=\"text/javascript\"></script><script src=\"//s1.hdslb.com/bfs/cm/st/bundle.js\" type=\"text/javascript\"></script><script type=\"text/javascript\">$(function(){\n",
       "        //日志上报\n",
       "        window.config = { sample : 1, errorTracker : false , resourceTracker: false , scrollTracker: true,   msgObjects : ''}\n",
       "        var jq = document.createElement('script');\n",
       "        jq.src = \"//s1.hdslb.com/bfs/static/log/report/reporter.js\";\n",
       "        document.getElementsByTagName('body')[0].appendChild(jq);\n",
       "      })</script><!-- built files will be auto injected --><script src=\"//s1.hdslb.com/bfs/static/phoenix/home/static/js/main.c14a219131f730a29e8e.js\" type=\"text/javascript\"></script></body></html>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "!!!!!!!!!\n",
      "Signal v. Noise - Medium\n",
      "Eschaton\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Google Blogoscoped\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Gizmodo\n",
      "Failed to parse feed\n",
      "The Official Google Blog\n",
      "Google Operating System\n",
      "Creating Passionate Users\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Joi Ito's Web\n",
      "Mashable\n",
      "Failed to parse feed\n",
      "The Viral Garden\n",
      "\n",
      "!!!!!!!!!\n",
      "Failed to parse feed\n",
      "PaulStamatiou.com - Technology, Design and Photography\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "O'Reilly Radar\n",
      "Failed to parse feed\n",
      "Scobleizer\n",
      "Seth Godin's Blog on marketing, tribes and respect\n",
      "Slashdot\n",
      "ThinkProgress - Medium\n",
      "The Dish\n",
      "WIL WHEATON dot NET\n",
      "43 Folders\n",
      "456 Berea Street\n",
      "Autoblog\n",
      "The Write News\n",
      "Failed to parse feed\n",
      "blog maverick\n",
      "Boing Boing\n",
      "BuzzMachine\n",
      "Captain's Quarters\n",
      "Failed to parse feed\n",
      "Copyblogger\n",
      "Latest from Crooks and Liars\n",
      "Daily Kos\n",
      "Deadspin\n",
      "Failed to parse feed\n",
      "Engadget RSS Feed\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Gothamist\n",
      "The Full Feed from HuffingtonPost.com\n",
      "Joho the Blog\n",
      "Joel on Software\n",
      "Failed to parse feed\n",
      "Kotaku\n",
      "kottke.org\n",
      "Lifehack\n",
      "Lifehacker\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "Matt Cutts: Gadgets, Google, and SEO\n",
      "MetaFilter\n",
      "mezzoblue\n",
      "Failed to parse feed\n",
      "Neil Gaiman's Journal\n",
      "Oilman\n",
      "PerezHilton\n",
      "plasticbag.org\n",
      "Derek Powazek\n",
      "ProBlogger\n",
      "Quick Online Tips\n",
      "Failed to parse feed\n",
      "Schneier on Security\n",
      "Failed to parse feed\n",
      "Search Engine Roundtable\n",
      "ShoeMoney\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "SpikedHumor - Today's Videos and Pictures\n",
      "Steve Pavlina\n",
      "Failed to parse feed\n",
      "ongoing by Tim Bray\n",
      "TechCrunch\n",
      "Techdirt.\n",
      "Failed to parse feed\n",
      "Failed to parse feed\n",
      "TMZ.com\n",
      "Failed to parse feed\n"
     ]
    }
   ],
   "source": [
    "import feedparser\n",
    "\n",
    "# Returns title and dictionary of word counts for an RSS feed\n",
    "def getwordcounts(url):\n",
    "    # Parse the feed\n",
    "    d = feedparser.parse(url)\n",
    "    wc = {}\n",
    "\n",
    "#     # Loop over all the entries\n",
    "#     for e in d.entries:\n",
    "#         if 'summary' in e:\n",
    "#             summary = e.summary\n",
    "#         else:\n",
    "#             summary = e.description\n",
    "\n",
    "#         # Extract a list of words\n",
    "#         words = getwords(e.title + ' ' + summary)\n",
    "#         for word in words:\n",
    "#             wc.setdefault(word, 0)\n",
    "#             wc[word] += 1\n",
    "\n",
    "    title = d.feed.title\n",
    "    print (title)\n",
    "    if title == '':\n",
    "        print('!!!!!!!!!')\n",
    "    return title, wc\n",
    "\n",
    "\n",
    "def getwords(html):\n",
    "    # Remove all the HTML tags\n",
    "    txt = re.compile(r'<[^>]+>').sub('', html)\n",
    "\n",
    "    # Split words by all non-alpha characters\n",
    "    words = re.compile(r'[^A-Z^a-z]+').split(txt)\n",
    "\n",
    "    # Convert to lowercase\n",
    "    return [word.lower() for word in words if word != '']\n",
    "\n",
    "\n",
    "apcount = {}\n",
    "wordcounts = {}\n",
    "feedlist = [line for line in open('data/feedlist.txt')]\n",
    "for feedurl in feedlist:\n",
    "    try:\n",
    "        title, wc = getwordcounts(feedurl)\n",
    "        wordcounts[title] = wc\n",
    "        for word, count in wc.items():\n",
    "            apcount.setdefault(word, 0)\n",
    "            if count > 1:\n",
    "                apcount[word] += 1\n",
    "    except:\n",
    "        print('Failed to parse feed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'wordcounts' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-421c3242e0da>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mwordcounts\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'wordcounts' is not defined"
     ]
    }
   ],
   "source": [
    "prin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
